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Another Look At "iht Power Of Meta-Analysis In The 
Solomon Fcur-Group Design 

Braver and Braver (1988) promoted a meta-analysis technique as the most powerful 
single test for analyzing data from a Solomon Four-Group Design. Customary 
analysis on fabricated data failed to find significance, but the mtra-analysis found 
significance. The current paper counters that meta analysis is a five-step process, 
and further, fabricated data sets can easily be constructed to show that customary 
tests can fina significance, whereas the meta-a.nalysis does not. Statements about 
the efficacy and power of meta-analysis should be made more cautiously. 
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Another Look At The Power Of Meta-Analysis In The 
Solomon Four-Group Design 

Recently, Braver and Braver (1988) made a strong case in Psychological Bulletin for 
reconsidering the under-used Solomon Four-Group Design (Table 1). They pointed out that 
perhaps its most serious drawback is the inability of the researcher to pinpoint the precise 
statistical treatment of the design. The data analysis solution proposed by Braver and Braver is a 
meta-analysis technique by Stouffer (see Braver and Braver for references;. 

They provided a flowchart to help visualize the five step process that leads to the 
meta-analysis. In brief, it begins with a 2x2 ANOVA (referred to as Test A) on O,, O^, 
and O^, the post test scores for the four groups. Failing to find significance, the flowchart 
proceeds to a main effects test on the experimental vs. control effect (Test D). If significant 
results are still not found, the process coiitinues with eithe . ANCOVA (Test E) on and 
with Oj and as covariates, a gain score analysis (Test F), or a repeated measures 
ANOVA (Test G), as appropriate. If the result at this phase is not significant, the flowchart 
moves on to a t test on and (Test H). Only when a nonsignificant result again appears 
does the flowchart bring the researcher to the meta-analysis (Test I). 

It should be noted that if significance is found at any of the four steps prior to the 
meta-analysis. Braver and Braver advised, "further testing may cease." Thus, meta-analysis is 
only performed when the customary statistical treatment fails to indicate a significant freatment 
effect. 
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The formula for the meta-analysis is 



^meta ^pi / v/ k , 

where Z^y is the Z value of the one-tailed p value of the Uh statistical test and k=number of 
tests. 

Braver and Braver said, "Test I allows full use of all data and thus is the most powerful 
cingle test of the treatment effect available." To demonstrate the power of Test I they con- 
structed a fabricated data set. The series of four customary tests failed to find significance. 
Then, they performed a meta-analysis on the last two tests (E and H), finding a significant 
result of p=.040. On this basis they concluded, "These hypothetical data show the superior power 
of the meta-analytic technique, because none of the customary analyses reached significance but 
the meta-analysis did." 

There are several concerns with Braver and Braver's conclusion. A minor objection the 
applied researcher may have is with the view of a five step meta-analysis as a single test, espe- 
cially since it cannot be done at all until at least tests E » H have been performed. Typically 
in meta-analysis the original studies were previously performed, requiring but a single method 
applied to those results. Second, meta-analysis techniques are usually employed with apparently 
diverse or seemingly conflicting results, as opposed to the situation when studies all reached the 
same statistical conclusion. Although in theory meta-analysis could be used when all of the 
studies are barely nonsignificant, we have never seen such a situation published. Braver and 
Braver's prescription for meta-analysis is restricted to the when all previous analyses reached 
the same (nonsignificant) conclusion. 



A more serious concern is promoting the meta-analysis technique as the most powerful 
test for the Solomon Four-Group Design. Although a most powerful test should find sig- 
nificance (if it exists) when weaker tests cannot, certainly where a weaker test finds sig- 
nificance, so too should a most powerful test. 

This paper will demonstrate that the meta-analysis technique can fail to find significance 
even though an earlier "weaker" test found significance. This is particularly disquieting, since 
Braver and Braver advised to cease testing at any step where significant results were obtained, 
thereby never allowing fo: this possibility to surface. 

Table 2 contains a set of fabricated data. Test E (ANCOVA) and H {V required by the 
meta-analysis, and Test I (meta-analysis) will be performed on the data. Table 3 reports sum- 
mary information for the data. Table 4 indicates that Test E (ANCOVA) was not significant, 
with p=.934. Table 5 shows the result for Test H (i), t = 2.07, with p = .048, a significant result. 

[Place Tables 1-5 about here] 
^meta 

I.98+.08 

^meta 

v'2 

1.4567 
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which yields a p of .072. As Braver and Braver said, it "is substantial but not quite significant 
by conventional standards". 

This set of fabricated data is not an exception to the rule. Any number of data sets can 
be constructed where one of the earlier tests is significant and the meta-analysis is not, by solv- 
ing for "pb" when 

^pa ^pb 

< 1.65 

^2 

where Z^^ refers to the Z associated with the p value of Test a, Zpj, is the Z associated with 
the p value of Test b, and pa < .05. So too, any number of data sets can be fabricated to show 
results like Braver and Braver obtained. 

If it is prudent to argue, as we do, that the use of a single fabricated data set is insuffi- 
cient evidence of meta-analysis being the most powerful technique in the Solomon Four-Group 
Design, is ■ equally prudent to say that a single fabricated data set is insufficient in showing 
the opposite, L'raver and Braver's innovative use of meta-analysis in the Solomon Four-Group 
Design should reawaken the interest of the researcher concerned with external validity issues. 
However, its efficacy and power in comparison to traditional analyses remains to be seen, and 
we advise using the technique with caution. 
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Table 1. 

Solomon Four-Group Design. 





Qrpup 




Treatment 


P<?Stt95? 


R 


1 








R 


2 






O4 


R 


3 






O5 


R 


4 






06 



Note. 0 - outcome measure; X = treatment: R = randomization 
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Table 2. 

Fabricated Data For Solomon Four-Group Design. 



0 



O: 



0, 



64 


54 


63 


54 


56 


50 


64 


59 


60 


62 


64 


50 


66 


55 


59 


62 


64 


51 


64 


56 


58 


51 - 


55 


51 


64 


55 


59 


57 


59 


51 


63 


65 


65 


56 


57 


61 


61 


64 


62 


56 


57 


60 


58 


64 


58 


56 


59 


61 


S8 


65 


60 


61 


62 


62 


59 


65 


58 


70 


71 


60 


57 


64 


66 


70 


71 


60 


62 


65 


63 


67 


68 


62 


65 


68 


65 


75 


76 


70 


66 


69 


58 


75 


78 


69 
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Table 3. 



Summary Statistics for Hypothetical Data. 




Correlation 






Pretest 


between pre- 




Postest 


Group iVf Variance 


and posttest 


M 


Variance 


1 62.2 9.6 




62.0 


26.1 




- 33 






2 61.0 8.6 




62.3 


62.2 




.18 






3 




64.1 


57.8 


4 




58.4 


46.1 



Note. . V = 14 per group. 
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Table 4. 



Test E: ANCOVA on 0^ 


and 0^ with Oj 


and Oj 


As dnvaritttp^ 




Source 


MS 


df 


F 


P 


Treatment vs. Not 


.321 


1 


.007 


.934 


Error 


1200.04 


25 







ERIC 



9 

12 



Table 5. 

Test H: ion 0^ and O^. 



df 



2.07 26 



.048 
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